Method and an apparatus for determining a gaze point on a three-dimensional object

ABSTRACT

A system for determining the gaze endpoint of a subject, the system comprising: a eye tracking unit adapted to determine the gaze direction of one or more eyes of the subject; a head tracking unit adapted to determine the position comprising location and orientation of the eye tracker with respect to a reference coordinate system; a 3D Structure representation unit, that uses the 3D structure and position of objects of the scene in the reference coordinate system to provide a 3D structure representation of the scene; based on the gaze direction, the eye tracker position and the 3D structure representation, calculating the gaze endpoint on an object of the 3D structure representation of the scene or determining the object itself.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/290,934, filed on Mar. 3, 2019, which is a continuation of U.S.patent application Ser. No. 14/428,608, filed on Mar. 16, 2015, which isthe national phase entry of Intl. App. No. PCT/EP2013/069236, filed onSep. 17, 2013, which claims priority to European Patent App. No.12184725.5, filed on Sep. 17, 2012, all of which are hereby incorporatedby reference in their entireties.

FIELD OF THE INVENTION

The present invention relates to a method and an apparatus for gazeendpoint determination, in particular for determining a gaze endpoint ofa subject on a three-dimensional object in space.

BACKGROUND OF THE INVENTION

There are existing solutions to the problem of finding the point or theobject or more specific the part of an object's surface that a (possiblymoving) person gazes at. Such solutions are described below and can besplit into separate parts.

At first the gaze direction of the person (or a representation thereoflike a pupil/CR combination, cornea center and pupil/limbus etc.) is tobe found.

For determining the gaze direction eye trackers can be used. EyeTrackers observe features of the eye like the pupil, the limbus, bloodvessels on the sclera, the eyeball or reflections of light sources(corneal reflections) in order to calculate the direction of the gaze.

This gaze direction is then mapped to an image of the scene captured bya head-mounted scene camera or a scene camera at any fixed location. Thehead-mounted scene camera is fixed with respect to the head, andtherefore such a mapping can be performed, once a correspondingcalibration has been executed. For performing the calibration a user mayhave to gaze at several defined points in the scene image captured bythe head-mounted camera. By using the correspondingly detected gazedirections the calibration can be performed resulting in atransformation which maps a gaze direction to a corresponding point inthe scene image. In this approach any kind of eye tracker can be used ifit allows mapping the gaze direction into images of a head-mounted scenecamera.

This approach enables the determination of a gaze point in the sceneimage as taken by the head-mounted scene camera.

As a next step it can be of interest to map the gaze point in the sceneimage as captured by the head-mounted scene camera, which can change dueto the movement of the subject, to a point in a (stable) reference imagewhich does not move and which corresponds to a “real world” object or animage thereof. The reference image thereby typically is taken from adifferent camera position than the scene image taken by the head-mountedscene camera, because the scene camera may move together with the headof the user.

For such a case where the head moves, there are known approaches fordetermining the gaze point in a reference image which does not movebased on the detection of the gaze direction with respect to a certainscene image as taken by the head-mounted scene camera even after thehead has moved.

One possible approach of determining the point gazed at is to intersectthe gaze direction with a virtual scene plane defined relative to theeye tracker. WO 2010/083853 A1 discloses to use active IR markers forthat purpose, which are fixed at certain locations, e.g. attached to abookshelf. The locations of these markers are first detected withrespect to a “test scene” which acts as a “reference” image obtained bythe head-mounted camera, by use of two orthogonal IR line detectorswhich detect the two orthogonal angles by detecting the maximumintensity of the two line sensors. The detected angles of an IR sourcecorrespond to its location in the reference image. Then the angles ofthe markers are detected for a later detected scene taken by thehead-mounted camera from a different position, thereby detecting thelocation of the IR sources in the later scene image. Then there isdetermined the “perspective projection”, which is the mapping thattransforms the locations of the IR sources as detected in an image takenlater (a scene image), when the head-mounted camera is at a differentlocation, to the locations of the IR light sources in the test image (orreference image). With this transformation a gaze point as determinedlater for the scene image can also be transformed into the corresponding(actual) gaze point in the test image.

The mapping of the gaze point from the actual “scene image” to a stablereference image which is time invariant becomes possible by defining theplane on which the gaze point is mapped in relation to scene stablemarkers instead of to the eye tracker (ET). This way the plane of thereference image becomes stable over time and gazes of other participantscan also be mapped onto it so that the gaze point information can beaggregated over time as well as over participants like it could only bedone before with eye trackers located at a fixed position.

For that purpose the prior art as disclosed in WO 2010/083853 A1 uses IRsources as artificial markers the locations of which can be detected byorthogonal IR line detectors to detect the angles of maximum emission.

The usage of using IR sources as markers for determining the transformof the gaze point from a scene image to a reference image is complicatedand inconvenient.

In the European Patent application no. EP11158922.2 titled Method andApparatus for Gaze Point Mapping and filed by SensoMotoric InstrumentsGesellschaft far innovative Sensorik mbH which is incorporated herein byreference there is described a different approach. In this approachthere is provided an apparatus for mapping a gaze point of a subject ona scene image to a gaze point in a reference image, wherein said sceneimage and said reference image have been taken by a camera from adifferent position, said apparatus comprising:

-   -   A module for executing a feature detection algorithm on said        reference image to identify a plurality of characteristic        features and their locations in said reference image;    -   a module for executing said feature detection algorithm on said        scene image to re-identify said plurality of characteristic        features and their locations in said scene image;    -   a module for determining a point transfer mapping that        transforms point positions between said scene image and said        reference image based on the locations of said plurality of        characteristic features detected in said reference image and        said scene image;    -   a module for using said point transfer mapping to map a gaze        point which has been determined in said scene image to its        corresponding point in said reference image.

This enables the implementation of gaze point mapping which does notneed any artificial IR sources and IR detectors. It can operate onnormal and unamended images of natural scenes taken by normalCCD-cameras operating in the visible frequency range. For a detaileddescription of this approach reference is made to European Patentapplication no. EP11158922.2.

But even with this approach it is only possible to map a gaze of amoving subject to a certain predefined static plane, however, thedetermination of a gaze endpoint at any arbitrary object in 3D space isnot possible.

It is therefore an object of the invention to provide an approach whichcan determine the gaze endpoint at any arbitrary three-dimensionalobject in 3D-space.

SUMMARY OF THE INVENTION

According to one embodiment there is provided a system for determiningthe gaze endpoint of a subject, the system comprising:

-   -   an eye tracking unit adapted to determine the gaze direction of        one or more eyes of the subject;    -   a head tracking unit adapted to determine the position        comprising location and orientation of the head and/or the eye        tracking unit with respect to a reference coordinate system;    -   a 3D scene structure representation unit, that represents a        real-world scene and objects contained in the scene by        representing the objects of the real-world scene through their        3D position and/or their 3D-structure through coordinates in the        reference coordinate system to thereby provide a 3D structure        representation of the scene;    -   a calculating unit for calculating the gaze endpoint based on        the gaze direction, the eye tracker position and the 3D scene        structure representation, and/or for determining the object in        the 3D scene the subject is gazing at based on the gaze        direction, the eye tracker position and the 3D scene structure        representation

By using a 3D representation, an eye tracker and a head tracker therecan be determined not only a gaze point on a 2D plane but also an objectthe subject is gazing at and/or the gaze endpoint in 3D.

According to one embodiment the system comprises a module forcalculating the gaze endpoint on an object of the 3D structurerepresentation of the scene, wherein said gaze endpoint is calculatedbased on the intersection of the gaze direction with an object in the 3Dstructure scene representation.

The intersection of gaze direction with the 3D representation gives ageometrical approach for calculating the location where the gaze “hits”or intersects the 3D structure and therefore delivers the real gazeendpoint. Thereby a real gaze endpoint on a 3D object in the scene canbe determined.

According to one embodiment the system comprises a module forcalculating the gaze endpoint based on the intersection of the gazedirections of the two eyes of the subject, and/or a module fordetermining the object the subject is gazing at based on the calculatedgaze endpoint and the 3D position and/or 3D structure of the objects ofthe real world scene.

By using the vergence to calculate the intersection of the gazedirection of the eyes of the subject there can be determined the gazeendpoint. This gaze endpoint can then be used to determine the objectthe user is gazing at.

According to one embodiment the object being gazed at is determined asthe object the subject is gazing at by choosing the object whose 3Dposition and/or structure is closest to the calculated gaze endpoint,

According to one embodiment said eye tracking unit which is adapted todetermining the gaze direction of the said one or more eyes of saidsubject is adapted to determine a probability distribution of said gazedirection of said one or more eyes, and wherein said calculating unitfor determining the object being gazed at determines for one or moreobjects the probability of said objects being gazed at based on aprobability distribution of gaze endpoints.

In this manner there can be determined a probability distribution whichindicates the probability that the subject gazes at a certain object.

According to one embodiment the system further comprises:

-   -   a scene camera adapted to acquire one or more images of the        scene from an arbitrary viewpoint;    -   a module for mapping a 3D gaze endpoint onto the image plane of        the scene image taken by the scene camera.

In this way not only the 3D gaze endpoint on the 3D structure isdetermined, but there can be determined the corresponding location onany scene image as taken by a scene camera. This allows thedetermination of the gaze point in a scene image taken by a camera froman arbitrary point of view, in other words form an arbitrary location.

According to one embodiment the position of the scene camera is known ordetermined by some position determination or object tracking mechanismand the mapping is performed by performing a projection of the 3D gazeendpoint onto an image of said scene camera.

This is a way of deriving from the 3D gaze endpoint the correspondingpoint in a scene image taken by a camera at an arbitrary location.

According to one embodiment the system further comprises:

-   -   A module for generating a scene image as seen from an arbitrary        viewpoint based on the 3D structure representation;    -   a module for mapping a 3D gaze endpoint onto the image plane of        the image generated by said scene image generating module,        wherein the mapping is performed by performing a projection of        the 3D gaze endpoint onto the image plane of said scene image        generated by said scene image generating module.

In this manner an arbitrary scene image can be generated not by takingan image using a scene camera but instead by generating it based on the3D structure representation. In this scene image then the gaze endpointor the object being gazed at can be indicated or visualized byprojecting the gaze endpoint onto the scene image or by e.g.highlighting the object which has been determined as the object of the3D structure being gazed at in the scene image.

According to one embodiment said eye tracker is a head-mounted eyetracker; and/or said scene camera is a head-mounted scene camera.

Head-mounted eye tracker and head-mounted scene cameras are convenientimplementations of these devices. Moreover, if the eye tracker ishead-mounted, then the head tracker automatically also delivers theposition/orientation of the eye tracker. The same is true for the scenecamera. Using the position (location and orientation) of the head asdetermined by the head tracker one can determine based on the gazedirection as determined by the head-mounted eye tracker in thecoordinate system of the eye tracker a corresponding gaze direction inthe reference coordinate system of the head tracker. This can be done bya simple transformation which transforms the gaze direction from the eyetracker's coordinate system into the coordinate system of the headtracker using the head location and orientation as determined by thehead tracker. The position delivered by the head tracker automaticallyalso delivers the position of the eye tracker through the given setup inwhich the eye tracker is fixed to the head and has a defined spatialrelationship with the head, e.g. by the mounting frame through which itis mounted on the head.

According to one embodiment said 3D Structure representation unitcomprises a 3D scene structure detection unit that is adapted todetermine the 3D structure and position of objects of the scene or theirgeometric surface structure in the reference coordinate system to obtaina 3D structure representation of the real-world scene.

In this way the 3D structure or at least the relevant, visible part ofit can be directly obtained from the scene by using the structuredetection unit.

According to one embodiment said 3D structure detection unit comprisesone of the following:

-   -   a laser scanner, possibly combined with a camera;    -   an optical scanner together with a light source emitting        structured light;    -   a stereo camera system;    -   an ultrasound detector;    -   any mechanical detection implementation.

These are convenient implementations of the 3D structure detection unit.

According to one embodiment the system comprises one or more of thefollowing:

-   -   3D gaze endpoints are mapped to one or more scene images taken        by a plurality of different scene cameras and/or to scene images        taken from different viewpoints;    -   3D gaze endpoints are mapped for a plurality of different        subjects to the same scene image;    -   3D gaze endpoints are mapped or aggregated over time to the same        scene image, possibly for different subjects.

This takes advantage of the flexibility of the approach by mapping thegaze endpoints for different users and/or for different scene cameras atdifferent locations. The recording of gaze endpoints and the mapping toone or more possibly different scene images can be performed over time,possibly even for different subjects, thereby obtaining a representationof the gaze data in a desired way.

According to one embodiment the mapped 3D gaze endpoints over time arevisualized in the scene image by visualizing the 3D gaze endpointstogether with the corresponding frequency of views or accumulatedviewing time, possibly distinguished according to different subjects.

This allows a visualization of the measured gaze endpoints and theirmapped scene locations.

According to one embodiment said visualization uses one or more of:

-   -   A heat map;    -   A focus map;    -   The center of gravity of gaze;    -   An automatic contour of viewing time.

These are suitable implementations for the visualization.

According to one embodiment said 3D Structure Detector repeatedlydetermines said 3D structure to enable a real-time gaze point detectionusing said eye tracker and said head tracker even if said 3D scene isnot static, or said 3D scene Structure Detector initially determinessaid 3D structure and an object tracker tracks the movement of one ormore objects in the scene to thereby enable a gaze point determinationover time using the tracked objects and the tracked gaze direction overtime.

In this way an online measurement can be implemented even for non-staticscenes.

According to one embodiment said 3D Structure detection unit comprisesone or more scene cameras and a computation unit for calculating said 3Dstructure based on said one or more cameras' images.

In this way the 3D-structure detection unit can be implemented withoutspecific hardware except a scene camera and a computation unit. Thescene camera(s) according to one embodiment may be the same scene cameraas is used for taking the scene image into which later the gaze endpointis to be mapped

According to one embodiment said computation unit uses a visual SLAM(visual Simultaneous Localization and Mapping) algorithm for calculatingsaid 3D structure and/or the position of the scene camera.

This is a suitable implementation of a 3D structure detection unit by ascene camera and a computation unit.

According to one embodiment the system comprises:

-   -   a display unit for displaying gaze data from one or more person        with a visualization of statistic data on the reference model,        wherein said visualization comprises:    -   a visualization based on a projection onto the surface of        objects;    -   a visualization based on a fly-through visualization of the 3D        structure.

These are suitable approaches for implementing visualizations of themeasured gaze endpoints.

According to one embodiment the images of the one or more scene camerasare combined to one or more bigger images such as a panorama or amultiperspective image to be used as scene image or images, and/orwherein said 3D structure representation unit uses the 3D structure andposition of objects of the scene in the reference coordinate system toprovide a 3D structure representation of the scene which has beendetermined in advance.

These are other suitable approaches for implementing visualizations ofthe measured gaze endpoints.

Using a 3D structure representation unit instead of a structuredetermination unit makes it possible to use 3D data which has beendetermined in advance.

DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B schematically illustrate an object representationaccording to an embodiment of the invention.

FIG. 2 schematically illustrates a gaze endpoint determination systemaccording to an embodiment of the invention.

FIG. 3 schematically illustrates a gaze endpoint determination systemaccording to a further embodiment of the invention.

DETAILED DESCRIPTION

In the following there will be described embodiments of the invention.

According to one embodiment there is determined a determination of agaze endpoint and in one embodiment also a mapping of the gaze endpointnot just for planes but for general objects in 3D-space. Moreover,according to one embodiment it can be determine which object in 3D-spacea subject is gazing at.

According to one embodiment for that purpose there is used a3D-structure detector, an object tracker for tracking the head of thesubject and its orientation (a “head tracker”) and an eye tracker. Bytracking the head movement using the head tracker and the gaze directionof the eye of the subject using the eye tracker there can be obtainedthe gaze direction of the subject in the 3D-space. This gaze directioncan then by projected or intersected with the 3D-model of the “world”which is obtained from the 3D-structure detector and viewed by thesubject. Thereby the point in the 3D-model to which the gaze of thesubject is directed and where it is “hitting an object” of the 3Dstructure can be determined. In this way the “gaze endpoint” and therebyalso the object at which the subject gazes at can be determined.

According to a further embodiment the gaze endpoint is determined basedon the vergence of the eyes. In this embodiment the eyetracker detectsthe gaze direction of the two eyes of the subject. When the subjectlooks at a certain object, then the gaze directions of the two eyes arenot parallel but they are directed to the same object which means thatthey intersect at the point of regard the subject is looking at. Thismeans that if the gaze directions of the two eyes are obtained by theeyetracker, the calculation of the intersection of the thus obtainedgaze directions in 3D-space actually provides the point of regard in 3Dspace.

It may happen that the two gaze directions which are determined for thetwo eyes of the subject do in fact not intersect at a certain point inspace. The reason may be that there is indeed no intersection, whichmeans that the two gaze directions indeed do not converge and intersectat the same point in space, or the lack of an intersection point may becaused by a measurement error. Nevertheless, in both cases there maystill be determined a gaze endpoint based on an intersection, e.g. bychoosing the point which lies halfway on the distance vector between thetwo gaze directions, in other words the point in 3D space which liesclosest to the two gaze directions.

By using the thus determined gaze endpoint and a representation ofobjects in 3D space it can then be determined which object the subjectis gazing at. This representation can be e.g. a full 3D structurerepresentation of the objects which are according to one embodimentobtained by a structure generation unit. The 3D structure representationof the objects define the structure of the objects (e.g. by theirboundaries). If the gaze endpoint determination is exact and withouterror, then it typically will lie on the surface of an object of the 3Dstructure, and this is then the point which is determined as the pointthe subject is gazing at. Once the gaze endpoint has been determined,from the 3D structure representation there also follows the object thegaze endpoint is lying on, and thereby the object the user is gazing at.

There may occur situations where the gaze endpoint does not lie on anobject. This may be due to different reasons, one being e.g. that thegaze endpoint determined by vergence is not fully correct and exact, andthen the thus determined gaze endpoint may lie somewhere in empty spacewhere no object is located. According to one embodiment, however, evenin such a situation there may be determined the object being gazed at,e.g. by determining the object which is closest to the gaze endpoint.This object may then be chosen as the one for which it has beendetermined that the subject is gazing at it.

Another approach for determining the object gazed at by the subject ischecking if both gaze vectors intersect with the volume of the object.In such a case, the object with which both gaze vectors intersect isdetermined to be the object at which the user is gazing.

According to one embodiment the gaze directions of the two eyes may beused to determine a “combined” gaze direction. This can e.g. be done byfirst calculating the gaze endpoint based on the vergence as theintersection of the gaze direction of the two eyes. The resulting gazeendpoint then can be used to determine a gaze direction which is basedon the gaze direction of the two eyes, in other words a “combined gazedirection”. This can according to one embodiment be done by choosing asgaze direction a vector which originates e.g. between the eyes of thesubject and passes through gaze endpoint which has been determined basedon the intersection. The resulting combined gaze direction can then beused for calculating its intersection with an object of the 3D structureto determine the object being gazed at.

According to one embodiment the eye tracking unit which is adapted todetermine the gaze direction of the one or more eyes of the subject isadapted to determine a probability distribution of the gaze direction ofthe eye or the eyes. This probability distribution may indicate for adetermined gaze direction a likelihood of being correct. It can e.g. beobtained based on the (known or estimated) accuracy or “errordistribution” of the eye tracker. This accuracy gives for a measurementvalue (i.e. the gaze direction) the probability that it is correctand—in form of a probability distribution—indicates for different valuesthe likelihood that they are the correct measurement value. Using such aprobability distribution one can—for the points in 3D space—indicate theprobability that they are lying on the measured gaze direction.

This probability distribution according to one embodiment is used todetermine for a plurality of objects their corresponding probability ofbeing gazed at. The probability distribution of the measured gazedirection corresponds to a probability distribution of different gazeendpoints. The probability distribution thereby may e.g. reflect theerror distribution of the measured gaze direction. E.g. if there ismeasured a certain gaze direction, then the error distribution indicatesthe different likelihoods of different gaze directions being correct dueto some measurement error (as indicated by the error distribution or“probability distribution”). Because for these different gaze directionsthere are resulting different gaze endpoints one can based on thedifferent gaze endpoints and their corresponding probabilities obtainthe respective probabilities of the corresponding objects being gazedat. This can e.g. be done by integrating the gaze probabilities of theindividual points which belong to the surface of an object over thewhole surface of this object. In this manner there is obtained a gazeendpoint probability distribution based on the gaze directionprobability distribution, and this is used to determine a probabilitydistribution which indicates for the various objects in 3D space theirprobability of being gazed at. In this embodiment the “calculating unitfor determining the object being gazed at” therefore actually determinesthe probability of an object being gazed as an implementation of the“determination of the object being gazed at”, in other words this is aspecific embodiment of a calculating unit for determining the objectbeing gazed at.

According to one embodiment the probability distribution of the gazedirection can be used also for the gaze directions determined for thetwo eyes of the subject. In this embodiment each of the two gazedirections has its own probability distribution which reflects thelikelihood of a certain gaze direction being correct. Based thereonthere can then for each point in 3D space be calculated the likelihoodthat this is the gaze endpoint as the intersection of the two gazedirections. In other words, this results then in a probabilitydistribution indication which for a point in 3D space indicates itsprobability of being the gaze endpoint. This probability distributionaccording to one embodiment is used to determine the probability of acertain object being gazed at; it follows directly from the probabilitydistribution of the gaze endpoints.

According to one embodiment, as mentioned before, the probabilitydistribution of the gaze direction reflects the “accuracy” or “error” ofthe measurement of the gaze direction. It may be determined by measuringthe error distribution or it may just be estimated. According to afurther embodiment not only the gaze direction is determined with acertain error being reflected by a probability distribution but also theposition of the objects in 3D space. For each object in 3D space thereis according to one embodiment an uncertainty of its accuracy which isreflected by a probability distribution with respect to the location ofthe object. This probability distribution can then be combined with theprobability distribution of the gaze direction(s) or the probabilitydistribution of the gaze endpoints to obtain a combined probability fora certain object being gazed at which reflects both uncertainties.

According to a further embodiment the objects need not to be representedby a full 3D representation of their shape, location and orientation.Instead, each object may just be represented by one representative pointin space which represents the location of the object in the 3D space.This representative point may e.g. be the center of gravity of the 3Dobject. Alternatively it may be any point, e.g. a user defined or userselected point, which represents the location of the object in 3D spacemay be chosen as representative point in 3D space which represents thelocation of the object. In this way the location of multiple objects maybe represented in 3D space. Based on the gaze endpoint determined byvergence there can then be determined the point which represents anobject which is most close to the gaze endpoint. In this way it may bedetermined that the subject is gazing at this object.

The object may also not just be represented by a single point, it may berepresented by some representative 3D representation which has someextension in two or 3 dimensions, e.g. by a plane area, or by a 3D shapelike a sphere which has a representative point as a center. One can useany space tessellation based on the scene objects here which can be usedto represent an object.

This is now illustrated in connection with FIGS. 1A and 1B. FIG. 1Ashows an example of a 3D real world scene. It includes a table and alamp hanging from the ceiling as real world objects. The 3Drepresentation of these objects is illustrated in FIG. 1B. The lamp isrepresented by a sphere having a center C in 3D space and a radius R.

The table is represented by a rectangle with its corners X1, X2, X3 andX4. The coordinates of X1 to X4 and C may be determined by somemeasurement to determine the 3D coordinates. The radius R may be chosensuch that it somehow resembles the shape of the “real lamp”.

This then results in a configuration as shown in FIG. 1B with twoobjects whose 3D location is represented by some representation in 3Dspace. Also shown is the head of the subject S. The position andorientation of the head in 3D space may be determined by some objecttracker (not shown), the gaze directions are obtained by someeyetracker, e.g. a head-mounted eyetracker (not shown). Using the gazedirections obtained by the eye tracker and the position and orientationfrom the head tracker (in its own coordinate system) one can thendetermine the gaze directions L1 and L2 in 3D space. This will later beexplained in even more detail.

Then the gaze direction intersection point G is determined as the pointof regard based on the vergence. As can be seen from FIG. 1A it does notlie on one of the object representations, neither on the lamp nor on thetable. Then there is determined the distance from the gaze point G tothe table and the lamp. It can be seen that the distance to the table D1is closer than to the lamp, and therefore the system then can concludethat the subject gazes at the table.

In this manner the system can determine the object a user is gazing atin 3D space.

According to another embodiment the 3D representation of the objectsuses a more accurate representation with higher granularity, e.g. a meshrepresenting the surface of the objects in 3D. In principle, however,the system then may operate in the same manner. If the gaze point isdetermined more accurately and lies on or near the surface of the 3Dobject representation, then the system may not only determine the objectthe subject is gazing at but even the location on the object the user isgazing at.

Now another embodiment will be described in somewhat more detail. Inthis embodiment the objects are represented by a 3D structurerepresentation, and the object the subject is gazing at is determinedbased on the intersection of the gaze direction with the 3D objectrepresentation rather than based on vergence.

In other words, with this approach based on a gaze direction which isdetermined and intersected with a 3D structure representing the “realworld” objects there can be determined a gaze endpoint on a 3D-object.According to one embodiment the thus determined gaze endpoint can bemapped to a corresponding location in any image of the scene. Moreover,in this way the problem of identifying the objects gazed at reduces tonaming the objects/object parts, since the approach directly deliversthe object the user gazes at because the gaze direction intersects withthe object the user is gazing at.

According to one embodiment a system for mapping a gaze onto a 3D-objectoperates as follows.

A detector to measure 3D Scene Structure (3D detector) is used todetermine the surface structure, position and orientation of all or allrelevant (e.g. selected ones or objects larger than a minimum size)objects in the scene resulting in a reference model (a 3D structure or“model of the 3D structure” of the “real world”). This reference modelis a representation of the “world” which the user gazes at. It consistsof a representation of the objects of the “world”, e.g. by a mesh.

It is e.g. represented in a “reference coordinate system”. The referencecoordinate system is time invariant and static, in contrary e.g. to thecoordinate system of a head mounted eye tracker which moves togetherwith the head of the subject.

The eye's position (the same applies to multiple eyes) can be measuredat any time in relation to the 3D Detector and in extension to thedetected scene objects by using a Head Tracker that relates the eye'sposition to the 3D detector position and/or the head tracker's position(and thereby gives also its position in the reference coordinatesystem). Preferably there is not only given the location but also theorientation of the head or a head-mounted eye tracker by the headtracker. Also preferably the head tracker coordinate system is timeinvariant as is the reference coordinate system of the 3D structuredetection unit. In one embodiment both coordinate systems are identical,in another embodiment there may be a time invariant transformation whichtransforms the head tracker coordinate system to the referencecoordinate system of the 3D structure detection unit or vice versa.

By combining the 3D detector and the Head Tracker with an Eye Tracker,that measures the gaze direction, the gaze intersection with surfaces ofobjects of the 3D structure can be calculated. The head-mounted eyetracker outputs the gaze direction in the coordinate system of thehead/eye tracker. Since the head position and its orientation is knownfrom the head tracker, the location and orientation of the eye trackeralso is known due to the known setup of the eye tracker being headmounted. Using this information from the head tracker the gaze directionin the reference coordinate system (the system in which the 3D structureis represented) can be derived based on the gaze direction determined bythe eye tracker in the eye tracker coordinate system by a simpletransformation of the eye tracker coordinate system to the referencecoordinate system. The transformation follows directly from the measuredlocation and orientation of the head measured by the head tracker.

This gaze direction can then be intersected with the 3D structurerepresentation of the scene to detect the 3D gaze endpoint on an objectof the 3D structure. Thus there is provided a measurement device thatmeasures the gaze endpoint of a person's (or a subject's) eye on 3Dobjects in the scene as well as parameters of the objects themselves.

This is a significantly novel approach for determining the gaze point.It is quite different from knowing the gaze point on images of the scenebecause in such a case objects have to be designated by hand for eachimage.

The approach extends over the previous approach of determining the gazepoint on a real scene plane by now detecting a 3D gaze endpoint on a 3Dstructure detected by the 3D-structure detector. Because the previousapproach using a scene plane operates only in a 2D space it does notcover points with parallax induced by object points off the plane, ofwhich there are usually plenty in real scenes. Therefore the presentapproach also overcomes such parallax problems.

Once the gaze endpoint on a 3D object has been determined the gazeendpoint can also be mapped to any image of the scene taken from anyarbitrary location by a camera. For that purpose the camera's parametersand its position relative to the scene are needed. They may be known bydefinition of the setup/calibration, or both can also be calculated outof the image itself given the scene structure, or they may be otherwisemeasured.

According to a further embodiment the scene image is not taken by ascene camera but instead is generated based on the 3D structurerepresentation, e.g. by projecting the structure into the image plane ofthe (arbitrary) scene image. Then in this arbitrary scene image theobject of the 3D structure which has been determined as the object beinggazed may be highlighted, or the gaze endpoint of the subject may bevisualized by projecting it from the 3D structure into the scene image.

According to one embodiment a user can name objects or even moredetailed object parts of the 3D structure by hand. In this way objectsmay be “tagged” with a name so that a “hit” of the gaze on such a namedobject then results in the return of the corresponding object nameAssuming the objects are the same over time, this has to be done onlyonce and the gaze on any object can be determined for all participantsand for all times any of the participants observed the scene. This isbecause the true 3D model of the object can cope with all possible viewswhich may be taken by a user.

For unnamed objects which are not manually tagged or labeled, the systemin one embodiment may assign default names.

For static scenes according to one embodiment the reference model can becreated offline. This is schematically illustrated in FIG. 2 . The3D-model/reference model is created “offline” using the 3D StructureDetector before the actual gaze measurement (this is illustrated as stepa) in the upper part of FIG. 2 . The 3D Structure Detector is not neededafterwards—the ET (eye tracker) and HT (head tracker) combination isthen sufficient to determine the 3D gaze endpoint on the 3D structurewhich was determined in step a). This is illustrated in the upper partof step b) illustrated in FIG. 2 which shows the determination of thegaze endpoint on the 3D structure.

Then the mapping of the gaze endpoint onto the scene image taken by ascene camera can be performed. For that purpose any 3D projection methodwhich maps the 3D structure to a 2D scene image using the position andparameters of the camera can be used. In this way the location where thegaze hits the 3D structure can be mapped onto the corresponding locationat a scene image taken by a scene camera. This mapping process isschematically illustrated in the lower part of step b) in FIG. 2 whichshows the mapping process (e.g. performed by using a 3D projection) ofthe 3D structure to a scene image.

The above approach works for static scenes. If one is interested indynamic scene content, according to one embodiment the 3D StructureDetector works in parallel to the ET and HT. This is schematicallyillustrated in FIG. 3 where the 3D structure is determined parallel tothe determination of the gaze by the ET, the head position of the HT,and the gaze endpoint mapping to a scene image.

According to one embodiment the dynamic change of a scene can be takeninto account by another mechanism. In this embodiment the 3D structureis determined only once, initially. However, the position andorientation of the relevant objects of the scene in the 3D space may bedetected and tracked over time by one or more object trackers. The gazedirection also is tracked over time. Based on the thus obtained trackingdata there can then be performed an offline processing which determinesover time the intersection between the gaze direction and the movingobjects and thereby determines the dynamic gaze endpoint.

Now there will be described a further embodiment with its componentsbeing described in somewhat more detail.

The components of this embodiment are gaze tracker, head tracker and 3Dstructure Detector. The gaze tracking can be realized by any of theconventional eye trackers. If a calibration is needed, the eye trackeris calibrated to a known plane in space so the gaze direction can becalculated from the gaze point on the plane.

As head trackers e.g. the following devices may be used:

-   -   A magnetic Head Tracker    -   Or an optical Head Tracker    -   Any kind of device that can measure the position and orientation        of the ET (or the eye itself) with respect to the 3D Structure        Detector (or the objects in the scene)

According to one embodiment there is used the scene camera incombination with the detected objects to calculate the scene camera'sposition and orientation. The camera may be the same scene camera as isused for taking the scene image into which later the gaze endpoint is tobe mapped. For the purpose of determining the camera position there maybe used a visual SLAM approach. A description of the visual SLAMapproach can e.g. be found in Andrew J. Davison, “Real-Time SimultaneousLocalisation and Mapping with a Single Camera”, ICCV2003, or in RichardA. Newcombe and Andrew J. Davison, “Live Dense Reconstruction with aSingle Moving Camera”, CVPR2010.

According to another embodiment the camera position may just bemeasured, e.g. by internal sensors of the camera (e.g. a GPS sensor), orit may be determined in some other way (e.g. by an Inertial MeasurementUnit or an object tracker).

According to one embodiment the ET position relative to the scene camerais known through the setup (both mounted on the same frame).

In order to determine the 3D structure several devices/approaches may beused.

Such measurement devices are e.g.

-   -   3D Scanners (Laser scanner, structured light scanner etc.)    -   Stereo camera system    -   Monocular camera system (e.g. visual SLAM)    -   Manual measurements

With respect to manual measurements, for example, the plans forbuildings are known in advance, e.g. from the construction plan, or theplans may have been derived by some “manual measurement”. The 3Dstructure detection unit may then consist in just a device for“obtaining” or “reading” the stored 3D structure data which has beenmeasured in advance, without performing an actual structuredetermination.

Instead of a 3D structure detection unit there may therefore be used a3D structure representation unit which uses the 3D structure andposition of objects of the scene in the reference coordinate system toprovide a 3D structure representation of the scene. The measurement ofthe 3D structure may have been carried out in advance, and the structurerepresentation unit then just uses the previously measured data toprovide the 3D structure representation.

According to one embodiment there is used a camera (e.g. the scenecamera) for static scenes to capture a video of the relevant scene partand calculate the scene structure by using a visual SLAM approach.Afterwards the approach allows also calculating the position of a camerataking an image of the scene from the image itself.

For dynamic scenes the structure can be measured online, which meansthat the 3D structure is repeatedly determined to take into account itsdynamic changes. Otherwise a combination of offline (or initial) 3Dstructure detection and tracking of the object(s) of interest (e.g. withthe help of an object tracker) can be used. For static scenes which donot change over time like supermarket shelves the structure can bemeasure once in advance.

The approach described before comes along with several advantages overexisting approaches as shown below.

Objects are unique over the scene and by extension so is the gazeendpoint onto objects. Classification of gaze can be done perobjects/object parts automatically if the classes of objects/objectparts have been defined, irrespective of the position of the user andthe scene image taken by the scene camera.

Gaze endpoints are decoupled from scene camera images. The gaze ismapped onto objects, not images of objects. For the ET with a scenecamera the gaze can be remapped from the reference model objects to theimage plane of the scene camera image. This is even true if the gazepoint falls outside of the scene image.

Because the gaze endpoint is calculated in a geometrically correct waythere is no parallax error in the gaze mapping, even if anon-parallax-free eye tracker is used.

It is possible to tell which object is actually gazed at in a scenewhere objects of interest are aligned behind each other such thatmultiple objects intersect the gaze path by using vergence. E.g. even ifthe gaze direction intersects with an object lying in front of anotherobject, but if the vergence based intersection of the two gaze pointslies on the object behind, then one may assume that the real gaze pointis the object behind.

New visualizations of gaze data is possible:

-   -   aggregated over time and/or participants    -   on object surfaces (e.g. heat map)    -   3D visualizations of the gaze rays in space, of the objects in        space, of objects    -   textured with mapped gaze data (heat map, focus map, etc.)    -   automatic contour, center of gravity etc. of objects projected        to a scene image    -   dynamic visualizations like fly throughs around the objects    -   aggregated gaze data on arbitrary scene images/on scene movies        (of which the movie from a participant's scene camera is one        special case)

In the following a further embodiment with its components will bedescribed.

First of all the system comprises an eye tracker that provides the gazedirection of a person relative to the coordinate frame of the head. Thegaze direction can also be defined indirectly as long it can betransformed to a head relative coordinate system.

Furthermore the system comprises a Head Tracker that detects the head orthe eye tracker's coordinate system's location and orientation relativeto a scene coordinate system. This can e.g. be done using sensors. Insome cases these sensors detect their own position relative to thescene, then the sensor would need to be head mounted. However, any headtracking device may be used.

Furthermore the system comprises a 3D Structure Detector that measuresthe three dimensional surface structure of objects. The structure ismade up out of the location, orientation and neighborhood of surfaceparts (points, patches, planes, or similar features used for describing3D structure). The detector may also measure appearance of the objects.

Optionally the system also comprises a scene camera (possibly combinedwith a position detecting device so its position and orientation isknown) that makes reference images of the scene.

Using these components there can be determined the gaze point on a 3Dstructure. Moreover, using the scene image and the scene camera'sposition there can be performed a mapping of the 3D gaze point onto thescene image.

Such a mapping can be performed onto any scene image given that thecamera position of the camera which takes the scene image is known, e.g.by using 3D projection of the 3D-gaze point onto the scene image.

According to one further embodiment the location of the gaze point in anarbitrary scene image can be performed in a slightly different way.Assuming that the gaze point at a first scene image has already beendetermined, then for a second scene image taken form a differentposition the gaze point mapping procedure as described in Europeanpatent application no. 11158922.2 can be used.

It should be noted that the 3D Structure Detector, Head Tracker, andposition detecting device can all be implemented by a camera combinedwith a suitable method to extract the necessary information out of theimages. In such an embodiment the eye tracker only needs to be combinedwith a scene camera and a device (such as a computer which is suitablyprogrammed) that carries out the extraction methods to extract the datasuch as the 3D structure, the camera position and the head position.

According to one embodiment, instead of a head-mounted eye tracker aremote eye tracker may be used. If this remote eye tracker is located ata fixed position and has a fixed coordinate system, its coordinatesystem may be used as reference coordinate system or it at least has aknown spatial relationship with the reference coordinate system of the3D structure detector. If the remote eye tracker is capable of directlyobtaining the gaze direction in its own time invariant coordinatesystem, then no further “separate” head tracker is needed, the eyetracker then—by the determination of the eye position and orientationwhich is performed also by such an eye tracker—simultaneously is also animplementation of a head tracking unit.

According to one embodiment the scene camera can move relative to theeyetracker. Its position may be determined by an object tracker and thenthe gaze point may be projected onto the scene image as described beforeregardless of its position.

What is claimed is:
 1. A method comprising: determining gaze directionsof two eyes of a subject via coordinates in a 3D reference coordinatesystem; determining an intersection of the gaze directions of the twoeyes based on a vergence of the two eyes upon determining the gazedirections converge; choosing a point lying closest to the gazedirections as the intersection upon determining the gaze directions donot intersect in space; determining a gaze endpoint in the 3D referencecoordinate system based on the intersection; and determining whichobject of the plurality of objects the subject is gazing at based on thegaze endpoint.
 2. The method of claim 1, further comprising: in responseto determining the object the subject is gazing at, displaying a tagassociated with the object the subject is gazing at.
 3. The method ofclaim 2, wherein the tag is received from a second subject other thanthe subject.
 4. The method of claim 1, further comprising: in responseto determining the object the subject is gazing at, receiving, from thesubject a tag; and associating the tag with the object the subject isgazing at.
 5. The method of claim 1, further comprising: in response todetermining the object the subject is gazing at, highlighting the objectthe subject is gazing at in an image of the scene.
 6. The method ofclaim 1, wherein the gaze endpoint lies in an empty space where noobject of the plurality of objects is located.
 7. The method of claim 6,wherein the object the subject is gazing at is determined by choosingthe object of the plurality of objects having a smallest respectivedistance between the gaze endpoint and the object, and wherein choosingthe object having the smallest respective distance is based on thevergence.
 8. The method of claim 1, wherein the gaze endpoint liesbehind an object of the plurality of objects other than the object thesubject is gazing at.
 9. The method of claim 1, further comprisingdetermining a location on the object the subject is gazing at based onthe gaze endpoint.
 10. The method of claim 1, further comprisingdetermining a probability distribution of the gaze endpoint anddetermining the respective probability of one or more of the pluralityof objects being the object the subject is gazing at based on theprobability distribution of the gaze endpoint.
 11. A non-transitorycomputer-readable medium having instructions encoded thereon which, whenexecuted by one or more processors of an electronic device, cause theelectronic device to: determine gaze directions of two eyes of asubject; determine an intersection of the gaze directions of the twoeyes based on a vergence of the two eyes upon determining the gazedirections converge; choose a point lying closest to the gaze directionsas the intersection upon determining the gaze directions do notintersect in space; represent a plurality of objects of a scene throughtheir 3D position and/or structure via coordinates in a 3D referencecoordinate system; determine a gaze endpoint in the 3D referencecoordinate system based on the intersection; and determine which objectof the plurality of objects the subject is gazing at based on the gazeendpoint.
 12. The non-transitory computer-readable medium of claim 11,wherein the instructions, when executed, further cause the electronicdevice to: in response to determining the object the subject is gazingat, display a tag associated with the object the subject is gazing at.13. The non-transitory computer-readable medium of claim 11, wherein theinstructions, when executed, further cause the electronic device to: inresponse to determining the object the subject is gazing at, receive,from the subject, a tag; and associate the tag with the object thesubject is gazing at.
 14. The non-transitory computer-readable medium ofclaim 11, wherein the instructions, when executed, further cause theelectronic device to: in response to determining the object the subjectis gazing at, highlight object subject is gazing at in an image of thescene.
 15. The non-transitory computer-readable medium of claim 11,wherein the gaze endpoint lies in an empty space where no object of theplurality of objects is located.
 16. The non-transitorycomputer-readable medium of claim 11, wherein the gaze endpoint liesbehind an object of the plurality of objects other than the object thesubject is gazing at.
 17. An electronic device comprising: an eyetracker to determine gaze directions of two eyes of a subject; a memoryto store a representation of a plurality of objects of a scene throughtheir 3D position and/or structure via coordinates in a 3D referencecoordinate system; and one or more processors to: determine anintersection of the gaze directions of the two eyes based on a vergenceof the two eyes upon determining the gaze directions converge; choose apoint lying closest to the gaze directions as the intersection upondetermining the gaze directions do not intersect in space; and determinea gaze endpoint in the 3D reference coordinate system based on theintersection, and determine which object of the plurality of objects thesubject is gazing at based on the gaze endpoint.
 18. The electronicdevice of claim 17, further comprising: a display to, in response to theone or more processors determining the object the subject is gazing at,display a tag associated with the object the subject is gazing at. 19.The electronic device of claim 17, further comprising: an input deviceto, in response to the one or more processors determining the object thesubject is gazing at, receive, from the subject, a tag, wherein the oneor more processors associate the tag the object the subject is gazingat.
 20. The electronic device of claim 17, further comprising: a displayto, in response to the one or more processors determining the object thesubject is gazing at, display an image of the scene with the object thesubject is gazing at highlighted.
 21. The non-transitorycomputer-readable medium of claim
 12. wherein the tag is received from asecond subject other than the subject.
 22. The non-transitorycomputer-readable medium of claim 15, wherein the object the subject isgazing at is determined by choosing the object of the plurality ofobjects having a smallest respective distance between the gaze endpointand the object, and wherein choosing the object having the smallestrespective distance is based on the vergence.
 23. The non-transitorycomputer-readable medium of claim 11, wherein the instructions, whenexecuted, further cause the electronic device to determine a location onthe object the subject is gazing at based on the gaze endpoint.
 24. Thenon-transitory computer-readable medium of claim 11, wherein theinstructions, when executed, further cause the electronic device todetermine a probability distribution of the gaze endpoint and determinethe respective probability of one or more of the plurality of objectsbeing the object the subject is gazing at based on the probabilitydistribution of the gaze endpoint.
 25. The electronic device of claim18, wherein the tag is received from a second subject other than thesubject.
 26. The electronic device of claim 17, wherein the gazeendpoint lies in an empty space where no object of the plurality ofobjects is located.
 27. The electronic device of claim 26, wherein theobject the subject is gazing at is determined by choosing the object ofthe plurality of objects having a smallest respective distance betweenthe gaze endpoint and the object, and wherein choosing the object havingthe smallest respective distance is based on the vergence.
 28. Theelectronic device of claim 17, wherein the gaze endpoint lies behind anobject of the plurality of objects other than the object the subject isgazing at.
 29. The electronic device of claim 17, wherein the one ormore processors determine a location on the object the subject is gazingat based on the gaze endpoint.
 30. The electronic device of claim 17,wherein the one or more processors determine a probability distributionof the gaze endpoint and determine the respective probability of one ormore of the plurality of objects being the object the subject is gazingat based on the probability distribution of the gaze endpoint.